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ABSTRACT- ' 

s To eliminate matur.atioif as a factor in the 

pretest-posttest desigiir pretest score's can be converted to 
anticipate pos1:test scores using grad^. equivalent scores from 
standardized tests. This conversion^ krcwn as historical regressio^, 
assumes tha t- without specific intejrventionr growth will continue at 
the riite (g rade , equivalents per ye^37, of schooling) ottaincd at the 
time of " pretest. Data VsX;e taken from lepcrts cf 213 Titl^ I ^ 
c ompejisa tory education programs iiv- fie » i'ork State to examine the 
predictive ability of the historical regression model. The approach 
was to: '(1) ^press historical regression in algebraic terms; (2) 
produce a linear model with assigned weights frcm the algetraic ^ 
,farttu^la; (3) produce a least sguares historical xegressicr model 
whose weights best fit the data; (4| ccBpate the historical 
regression model with the least sguares model; and. (5) develop an 
a^lternative model. H^en compared with program- lev^^l data, historical 
'regression underestimated fipal achieveoeiit for short programs with 
older children. It overestimated fbr younger children in long ^ 
programs. An alternative method was developed which eliminated -the 
biasr' removed half of th^ errot, and^ eliminated much comput-'aticn 
since an expected achievement level fcr^each child Has net required. 
(Aiithor/CP) ' . / 
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Abstract 



An Analysis of the Historical Regression Method of Predicting PoBtteet 
Grade EqulvalontB for Catogoi^ica I ly-Aidod Programo 
THOMAS L. HICK and DAVID J. IRVINE 
New York State Education Department 

Historical Regression follows directly from the assumption that, 



without specific intervention, growth will ef^Fttihue^at the rate igr-ad^ 
equival|/it8 per year of schooling) obtained at-the time of the pretest. 
When compared with program- level data (n =-'213)''it was found that 

Historical RegressioV underestimated final achievement for short 

t ' ^ 

programs with older children. It overestimat^ed for younger children 
in long prograims. An alternative method wa^s developed which e,l 1ml na ted 
' the blaB. removed half of the error, and eliminated much computation since 
an expected achievement level for each child was not required. 
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An Analysis: of the Historical Regression 
tiethod of ;Bredlctlng; Posttest Grade ^ 
Equivalents , for Categories lly-Alded ■ 
Programs ,i . 

!• h. WCK and p; j'. IRVINE 

New Y6i:k St:ate\Educatlon Department' 



' T. L. >llck 
Preklndergarten Evaluation Unit 
State Education Department 
Room '562 EBA ' , 
Albany/ New York 12234 



Objectives , " ■ ' * 

^ To eliminate maturation as a factor In the pr^test-posttest design 

at least one State '(^ew- York) ^ecommendsfa conversl^on of tfie pretest^ to 
antloipate ppsttest scores when data are '.In grade Equivalents from 
standardized' tests. This conversion Is known as HlstorfcJal Regression, 

'and follows directly fVom the assumption that, wlt^iout ^specif Ic ^Inter- 
ventltn, growth will continue at the rate (grade equivalents per year of- 
schooling) obtained at the time of pretest. i • . 

. The District Evaluator's Handbook describes tffi^ procedure for 
obtalnlng-flntlclpated scores by the following steps: 

Obtain each pupil's pre^'est Igrade equivalent. 
Subtract 1 (since most standardized tests start at 
1.0). / * 

J Step ^3. .Divide the figure obtained In step 2 by the number 
of months tftie pupil has been In school to^ obtain 
a hypothetljp^l (historical regreefslon) rate of 
growth per month. (Ignore kindergarten months. 
On^ school year = 10 months.) 
Step 4. Multiply ,the^. number of months of Title I treatment 
^ " by the hls^torlcal rate of growth per month.*' 

Add the figure obtained In step 4 to the pupil's 



Step 1. 

st^p 

Step 3,. 



Step 5. 



pretest gi^de equivalent (step 1).* 



This paper examines the Historical Regression method for. obtaining 
predicted posttest grade equivalents to determine its adequacy as a , 
predictive model and to d^elop an alternative predictive model.' 



Method 



The approach was to (1) express Histoi^cal Regression in algebraic 
terrn^, (2) from the algebraic formula, produce a linear model for 



^Bureau of Urban and Community Programs Evaluation. District Evaluator's 
Handbook of Selected Evaluation Procedures for Categorically Aided 
. Programs Serving. Disadva-nta^ed Lefl^raetrC Albany; New York State ' 
Education Deparjtment, Spring 1972. 
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Historical Regression (this rnod^l bias assumed weights) , (3) produce a 
Least Squares Historical' Regt^ssion model ' that has * weights which beet ; 
fit the data, (4) compare the;^Historical Regression model with the 
Loast Squares Historical ' Rc^rfSBion model to dotormino slmilarlticB 
and differences, (5) devel an a Iter native model . 

Data, Source ^ 

The data were taken from, ne reports of Title I Compensatory Educa- 
tion programs filed at the -New York State Education Department for the 
1972-73 school year. This file contained the following information for 
213 programs: 

Y - Program mean reading gi;ade equivalent on' posttest . 

B - Program mean reading grade equivalent on- pretest If over 

grade equivalent of 2. 
D - Duration of program in years, " 
. T - Mean previous time spent in schpdl in years. ' 

Characteristics of these variables are reported. In Tab]^,e 1. 



Table 1 ! 
Means, Standard .Deviations and Limits 



Variable 




M 


SD 


1 X ' •. 
Minimum 


Maximum- 


Y. 
B 
p 

nr 




4.67 
3.83 , 

.90 
6.69 


1.34 
1.23 
.32 
5.72 


2.10 ' 
.2.03 
.20 

' 1.00 : 


9.2 

■ > -7.7 
• .. 1.6 
11.0 


Is the Historical. 


i 

Regression 


alfiorlthm adequate? 





In terms of the defined variables the algebraic expression of^^e 
Historical Regression was found to be : ' 

Y ° B + (B -1) X (D/T) 

■•^n the form of a linear model this Historical Regression Model became: 

Model 1. Y «=> Zero U + 1 (B) + 1 (B x D/T) -1 (D/T) + -E^ > * 
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■ . ■ / ■ ' ■ ' ■ ' - ■ ■ ■ 

The weight?8 on the variable^ • were dldtatc^d by the His'toricdl 
Regression procedure. A model with the ^ame vartable^s used in the 
•'Historical Regression model but with tH'fe. weights lett fot a least 
oqudrGB boat fit waa Qxproaaed aa a Leaat SqUoroa^ Hiatorica 1' 
Regression Model: ,, 

Model 2. Y = a 0 + a, (B) + a^ (B k D/T) + a" (D/T) + E 
0 1 2 .32 

* ' ■ 

^The solution to the Ljcast Squares Historical Regression model 
yleld^' Qq = .68/ai = .99, a2 = .59 and = -1.02. The Historical ' 

Regression Model assumes that these weights are: - o, aj^ = 1, a2 = 1, 

a^ = jpl . The errott sum of squar^es foT> the'^Lcast Squat^es mode L Ls 43. ^ 

Upon Substitution o£\the histprical regression values, the error sum* 
; 'of squares for the *restricted model is 85. • ^-iith 5 decrees of freedom 

for the\full model an^ none for the restricted mpdel, the difference « 
' between the error sum of squares is 'tested with F at 5/208 degrees of 

freedom .^^^The F of 38 is asjodiVted with a near zero probability of a 

chance difference. . 

' * Therefore , the Historical Regr^ssTon Model (1) does a poorer job 
of predicting posttest grade equivalents than the Least Squares 
Historical Regression Model (2), , - 

Since the Historical Regression deviates from a Least Squares best 
fit. Historical Regression was plotted as shown in Figure 1, It may be 
observ^ed tha J: ^Inltia 1 grade equiva lent s are along the horizontal axis ^. 
while the ratio of the progjram duration to the average time the studenta 
had proviouHly Fipont in Rch*^^l Hs given on the right- side of the cube. 
The hiptorical rogrofision plane twists so that the relationship of pre- 
test to posttest is 1 to 1 as the ratio D/T approaches zero . This 
would be the situation of a very short*' program (e. g. , 2 months) for 
children that had spent a long time in sehool (e^g.*, 50 mo^nths) yielding 
a ratio of D/T pf .04, a value vfery close to zero. In this situation, 
according to historical regression, the childifen Will have the same score 
at the end of program as at the^ beginning. ^ ' 

I At the other end of the D/T ratio, the situation is sho\3tn wtt^re the 
program has lasted as .lotig as the child has previously^.been in school. 
This would be the caf^e^oV young children. At an initial grade equivalent- 
of 2, the final achieVement would be approximately 2.4 grade eqi ' 
At an initial grade equivamYit of 3, the^fiual achievement would x.^ 
approximately 4.0 grade equivalents. -^^^^^ ^ ^ 

This historical model does -^4% the data quite ^ell covering 88 / 
percent of the variability in posttest scores, provided it is centereci 
and twisted properly. As previously sItxjW^ however, the historTcaTJ^ 
regression model does not match the Least squares fit tb the data. 

■ . ' ^ - ■ 
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A Detailed Kxaminotion of the Ffl storlca 1 ^RcRrcsslDn Algorithm , 

: : ' 

Each of the Historical .Regression weights Was tested* in turn. with 
olthtir tho hypothonlsj'.od wotRht 'or . tho lonnt Bquaraw w<itght tctnlno^ In 
accord with the result of tes^ting-i th^ historical regression weight. 
First, the Qq = 0 hypothesis was 'tested. This hypothesis is^that the 
regression plane passes through zero or\ the posttest when the pretest - 
is zero and the D/T ration i^ zero . this turned out to not .be the^cfase. 
The least squares regression plane does not pass through -zero at a very ' 
high level of probabi lity* (J* = 476 a.t 1/209 degrees. of freedom) . The^ 
least squares value elevates the plane .68 « ^ , 

grade equivalents above zero.. .The^ second hypothesis v'as^-that a2 = 1. 
This 'weight describes the Amount of twist in the plane. ^ The weiglit for 
twiat was nat one (P = 0). weight of .59 fits the data much better 
so that the twist' , is not as large as assumed^y historical regreJssion. , 
The third hypothesis was that the tM't .of the >lane was/f5 degree^ whenw 
D/T was zero. This was the hypothesis that, ai was equal to^^ne. - Test V 
of tl^is hypothesis failed, to reject it, so that a value of One for a^ . 
was an acceptable f it ' by. , least squares (P = .28). The-last hypothesis 
that ao is -1^ also w^s not rejected (P '= .37). 

' The final'model is: ' . 

, . :^ • ^ . 

Y = .68 + B + ((.59B) .-Lf x D " , ' 

The differences between this equation and t^- algorithm of historical 
regression are the /addition of .68 to all projected scores , and the 
mul tiplica tl^on o& (Ihe pretest score by .59 before adjusting it for the % 
beginnj(.ng at grad^l. ^ \ V 5 

Examination of c the difference between tlve Historical R^egression 
plane in Figure 1 and t^eH.east sgjuares be^t rit plane reveals that the 
final achievement oT^^^^der students in short dura tion prograins has been 
underestimated by as much as' 1/2 a year. ^Younger students ^ long r^^o- 
grams did not fare as well. The effect was *^o overestimate uueir 
aqhievemoTit by as much as one year, gi^ng the appearance of ^poor ^ 
perf ormatice« ^ , i'^ '.^ 

. . V f " 

Development of ian Alternative Model 

It would se^jn desircTblc tOABdmpli^Ey the formulation -^-^e expected 
achievement' without losing^yie /predictive power of historica iN^egression. 
An attempt a^ this goal w^j^jf^jsi^^ta^ beginning with tjhis possible model: 

Y ° a^JJ a^B + a^D + a^B^ + a^D^ + a3B^ + a^D^ + 

ayixD + a^B^D t^a^BD^ + a^^B^D^ + 
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" JThe for^the abo^ model and is obviously significant, 

Thfe. hypothesis of'no curvilinear interaction, was rj^Jected using Model 4; ' 

which yielded an of .890. The difference between R'^'s was s^igni'f icantr 
iF^ = 4.12^ df 3/202, .008)-, therefore curvi linear interacti^'on was 

'considered as present. The task of locating thc^ interact ion remained. 
The hypotheses that the interaction was very complex was testqd with 
model 5 which vms the same as model 3 except ^or the last term B D . . . 
Complex interac&Lon was cori^id^red noti present (F,= 4.^57., df^ 1/202, 

'P = .03) as all relationships are testdd at the .Ol'level of significance. 
Therefore,*" mod el 5 becarje the full model for a- test of the hypiothesls 
that the interaction is curvilinear on program duration. Mgdel 6 
expressed this hypotljesis: ^ " ' - 

" ' ' 2 ^2 . T,3 ^ - 

^ Y = a^U + a^B + a^D + a^B + a^D + a5B + a^ 

a BxD + ag&^D + ' ^ 

This mddel turned out an R^ of) .889 which was significantly different 
from model' 5 at the .wO©7 level of significance <^ = 7.51 , dt = 1/203). 
Therefore,, the term BoKwafS Considered significant and included' in the 
*next modei (S) that* was used to test 'the effect of\ B^D. This term was 
'-n^t considered significant CR^ = .692, F = 2 . 64, ' d£ 1 /203 , P = ..10). 




To -regain our„>ehrings , the full mG^del is no^(#7) which includes 
:d degree polynomial forms: ^ * 

Y = a U + a^B + a^D + a«B^ '^'^iP'^ ^s? ^6^^ ayBxD 



a BD^ + E-^ 
9 7 



Con)toarison of^ this model witbl a model that lacked the third, degree 
polynomi'^ls showed that' B*^ and were not significant '(F = .80, df 2/204 
.p ="'.*45). .The only rema ining . term not involved in the established int^jpr- 
action was B^-. It was found to be nonsignificant (F - 1 df = 1/206) 

and .dropped , from the f'i'nal equation. 



- a All the iremaining terms were involved in the expression of the inter 
action and were therefore considered neces^Jia&y foi^the ex'pressioh of the* 
relationship of beginning scores and program duration on posttest scores. 
The f^nal acceptable model is: - 

■ \^ 

Y=-2.14 +1.56B +7.26D -4.04D^ .1.39Bxp •f.80BD^ 
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This is a highly' complex plaw as it describes a different effect 
for programjdurcMlon at various levels of the pretest scores.^ It 
should be noted that program duration, although highly slgnlf leapt 
hoB an only very subtlo rolotlot^shlp with poBttcat Bcorda after Initial 
skill level Is' Included. This ^lane described by the final acceptable 
model. Is stiown In Figure 2. -* ' ' 

Conclusloiv . . - ^ ' ^ 

A method fo^^xiUminatlng maturflti^n as a factor In pret^st- 
pysttest designs fW Title. I' programs was ^xamlned In nelatlWi to 
cyita and an alternative predictive model was developed^ The alterna- 
tive method: (I) makes no assumptions regarding the relationship 
between pretest level and prograi^l duration, (2) does not require the 
computation of , time span in school fd^ each child, (3) does not requir^ 
an expected achievement level for each chil^T, (4) does not bias 'againat 
some programs, (5) "cuts the error in half, (6) requires only the \ 
beginning mean achievement- level. ' . 
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• FIGURE 2 



LEAST- SQUARES MODEL FOR USING P,RETEST LEVEL AND PROGRAM 
DURATION TO ESTIMATE FINAL ACHIEVEMENT FOR THE PROGRAM. 
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